Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus includes: a reduced image generation unit configured to obtain a reduced image; a gain data generation unit configured to obtain gain data; and a determination unit configured to determine a control gain required to amplify the input image based on the generated gain data, and letting first gain data be the gain data for an image having a relatively high frequency, and second gain data be the gain data having a relatively low frequency, the determination unit determines the control gain based on a first addition result and a second addition result.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique required to control tonecharacteristics by applying different gains to respective portions of aninput image.

2. Description of the Related Art

Conventionally, a digital dodging process for controlling tonecharacteristics by applying different gains to respective portions of aninput image has been proposed. For example, this dodging process isapplied to a so-called backlight scene in which the brightness of a mainobject to be shot is considerably lower than that of a background. Inthe backlight scene, when the brightness of an entire image iscontrolled to an appropriate level, the main object is generally shotdarkly. Therefore, using luminance-dependent gains intended to apply arelatively large gain to a dark portion, the darkly shot main object iscontrolled to an appropriate brightness. The aforementioned process isthe basic concept of the dodging process.

Furthermore, as an input luminance upon calculation ofluminance-dependent gains, a low-frequency image generated fromluminance components of an input image is used. This is because when aninput image is used intact as an input luminance used to calculateluminance-dependent gains, a large gain is unwantedly applied to everydark object portions and black object portions in an image, and acontrast of an image after a gain process is considerably impaired.Therefore, using the low-frequency image, the influence of local darkportions in an image is reduced, and gain sensitivity to luminancechanges of an image lowers, thus executing the dodging process whilemaintaining a resolution of an output image.

However, when the dodging process is executed using the low-frequencyimage, a pseudo edge is generated near edge portions between alow-luminance region and high-luminance region, as shown in FIG. 25. Inthe example shown in FIG. 25, a small blur signal and large blur signalas low-frequency components of a source signal are used in addition tothe source signal. When a gain table has a pattern which is decreasedlargely relatively at a low-luminance side, as shown in FIG. 25, a gainconverted from the blur signal is largely decreased on the low-luminanceregion side of an edge portion. This is because a value of the blursignal of the edge portion becomes large due to the influence of thehigh-luminance region side. Therefore, a black-rimmed pseudo edge isgenerated, as shown in FIG. 25. This phenomenon occurs dominantly as arange of a gain to be multiplied (a difference between GAIN_MAX andGAIN_MIN) becomes larger.

To solve the aforementioned problems, for example, techniques describedin Japanese Patent Laid-Open Nos. 2010-244360 and 2009-272983 have beenproposed.

In Japanese Patent Laid-Open No. 2010-244360, gains are calculated froma plurality of luminance images having different frequency bands. Then,when a gain calculated from a relatively low-frequency image exceedsthat calculated from a relatively high-frequency image, the contributionof the gain calculated from the relatively low-frequency image iscontrolled to be reduced. More specifically, a use ratio of the gaincalculated from the high-frequency image is increased.

In Japanese Patent Laid-Open No. 2009-272983, images having differentfrequency bands are used as in Japanese Patent Laid-Open No.2010-244360, an image is divided into regions based on luminance valuesof a relatively high-frequency image, and a gain value calculated from arelatively low-frequency image is corrected by a region division result.

However, Japanese Patent Laid-Open No. 2010-244360 merely describes ameasure against a case in which a gain calculated from a low-frequencyimage becomes relatively large, that is, a measure against pseudo toneslike a white-rimmed high-luminance region, and cannot cope with pseudotones of a low-luminance region. Also, when a gain to be multiplied withthe low-luminance region becomes large, the method of Japanese PatentLaid-Open No. 2010-244360 cannot provide a sufficient effect, and pseudotones are left unremoved.

Japanese Patent Laid-Open No. 2009-272983 does not describe a measureagainst a case in which gains having largely different gains are mixedin a single region, as shown in FIG. 25, and a gain to be re-calculatedin that region is influenced by the largely different gains.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of theaforementioned problems, and suppresses generation of a pseudo edgewhile maintaining a satisfactory contrast of an output image when tonecharacteristics are controlled by applying different gains to respectiveportions of an input image.

According to the first aspect of the present invention, an imageprocessing apparatus comprises: a reduced image generation unitconfigured to obtain a reduced image by reducing an input image; a gaindata generation unit configured to obtain gain data by associatingluminance values for respective pixels with gains in association with aplurality of images, which include the reduced image and have differentreduction ratios; and a determination unit configured to determine acontrol gain required to amplify the input image based on the generatedgain data, wherein the determination unit determines the control gainbased on a first addition result as an addition result of gainscorresponding to pixels of interest of first gain data and second gaindata, and a second addition result as an addition result of a gaincorresponding to the pixel of interest of the first gain data, and again based on gains corresponding to pixels of a surrounding region ofthe pixel of interest in the second gain data, wherein the first gaindata is data for an image having a relatively high frequency of theplurality of images and the second gain data is data having a relativelylow frequency of the plurality of images.

According to the second aspect of the present invention, an imageprocessing apparatus comprises: a reduced image generation unitconfigured to obtain a reduced image by reducing an input image; animage addition unit configured to add a plurality of images whichinclude the reduced image and have different reduction ratios; and adetermination unit configured to determine a control gain required toamplify the input image based on luminance values of the added image,wherein the image addition unit performs weighted addition of a firstaddition result as an addition result of pixel values of pixels ofinterest of a first image and a second image, and a second additionresult as an addition result of a pixel value of the pixel of interestof the first image, and a pixel value based on pixel values of pixels ina surrounding region of the pixel of interest in the second image,wherein the first image is an image having a relatively high frequencyof the plurality of images and the second image is an image having arelatively low frequency of the plurality of images.

According to the third aspect of the present invention, an imageprocessing apparatus comprises: a gain data generation unit configuredto obtain gain data based on an input image; a reduced data generationunit configured to obtain reduced gain data by reducing the generatedgain data; and a determination unit configured to determine a controlgain required to amplify the input image based on a plurality of gaindata which include the reduced gain data and have different reductionratios, wherein the determination unit determines the control gain basedon a first addition result as an addition result of gains correspondingto pixels of interest of first gain data and second gain data, and asecond addition result as an addition result of a gain corresponding tothe pixel of interest of the first gain data, and a second additionresult as an addition result of a gain corresponding to the pixel ofinterest of the first gain data and a gain based on gains correspondingto pixels in a surrounding region of the pixel of interest in the secondgain data, wherein the first gain data is data having a relatively highfrequency of the plurality of gain data and the second gain data is datahaving a relatively low frequency of the plurality of gain data.

According to the fourth aspect of the present invention, an imageprocessing apparatus comprises: a band-limited image generation unitconfigured to generate, from an input image, a band-limited image havinga frequency band different from the input image; a gain data generationunit configured to obtain gain data respectively for a plurality ofimages which include the band-limited image and have different frequencybands; and a determination unit configured to determine a control gainrequired to amplify the input image based on the generated gain data,wherein the determination unit determines the control gain based on afirst addition result as an addition result of gains corresponding topixels of interest of first gain data and second gain data, and a secondaddition result as an addition result of a gain corresponding to thepixel of interest of the first gain data and a gain based on gainscorresponding to pixels in a surrounding region of the pixel of interestin the second gain data, the first gain data is data for an image havinga relatively high frequency of the plurality of images and the secondgain data is data for an image having a relatively low frequency of theplurality of images.

According to the fifth aspect of the present invention, an imageprocessing apparatus comprises: a band-limited image generation unitconfigured to generate, from an input image, a band-limited image havinga frequency band different from the input image; an image addition unitconfigured to add a plurality of images which include the band-limitedimage and have different frequency bands; and a determination unitconfigured to determine a control gain required to amplify the inputimage based on luminance values of the added image, wherein the imageaddition unit performs weighted addition of a first addition result asan addition result of pixel values of pixels of interest of a firstimage and a second image, and a second addition result as an additionresult of a pixel value of the pixel of interest of the first image anda pixel value based on pixel values of pixels in a surrounding region ofthe pixel of interest in the second image, wherein the first image is animage having a relatively high frequency of the plurality of images andthe second image is an image having a relatively low frequency of theplurality of images.

According to the sixth aspect of the present invention, an imageprocessing apparatus comprises: a gain data generation unit configuredto obtain gain data based on an input image; a band-limited datageneration unit configured to generate band-limited data having afrequency band different from the generated gain data; and adetermination unit configured to determine a control gain required toamplify the input image based on a plurality of gain data including theband-limited data, wherein the determination unit determines the controlgain based on a first addition result as an addition result of gainscorresponding to pixels of interest of first gain data and second gaindata, and a second addition result as an addition result of a gaincorresponding to the pixel of interest of the first gain data, and again based on gains corresponding to pixels in a surrounding region ofthe pixel of interest in the second gain data, wherein the first gaindata is data having a relatively high frequency of the plurality of gaindata and the second gain data is data having a relatively low frequencyof the plurality of gain data.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the arrangement of an image processingapparatus common to respective embodiments of the present invention;

FIG. 2 is a view showing a backlight scene assumed in the firstembodiment;

FIG. 3 is a graph showing an overview of luminance-dependent gainsassumed in the first embodiment;

FIG. 4 is a view showing a contrast drop caused by theluminance-dependent gains;

FIG. 5 is a block diagram of processes in a gain image generation unitaccording to the first embodiment;

FIG. 6 is a flowchart showing processes in a blur processing unit;

FIG. 7 is a view showing the process of the blur processing unit;

FIG. 8 is a block diagram showing processes of a gain additionprocessing unit;

FIG. 9 is a view showing the processing contents of an additionprocessing unit 2;

FIG. 10 is a graph showing the processing contents of the additionprocessing unit 2;

FIG. 11 is a flowchart showing the processing contents of the additionprocessing unit 2;

FIG. 12 is a view showing advantages of an addition processing unit 1and the addition processing unit 2;

FIG. 13 is a block diagram showing processes of a gain image generationunit according to the second embodiment;

FIG. 14 is a block diagram showing processes of a luminance additionprocessing unit;

FIG. 15 is a graph showing the relationship between a luminancedifference and gain difference;

FIG. 16 is a block diagram showing operations of a luminance→weightcalculation unit;

FIG. 17 is a graph showing a calculation of a gain amplification ratio;

FIG. 18 is a graph showing an operation of a luminance→weight conversionunit;

FIG. 19 is a block diagram showing processes of a gain image generationunit according to the third embodiment;

FIG. 20 is a flowchart showing the sequence of blur processing accordingto the third embodiment;

FIG. 21 is a graph showing a calculation of a luminance amplificationratio;

FIG. 22 is a block diagram showing processes of a gain image generationunit according to the fourth embodiment;

FIG. 23 is a block diagram showing processes of a gain image generationunit according to the fifth embodiment;

FIG. 24 is a block diagram showing processes of a gain image generationunit according to the sixth embodiment; and

FIG. 25 is a view showing a problem (generation of a pseudo edge) of therelated art.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described in detailhereinafter with reference to the drawings. Note that embodiments to bedescribed hereinafter are presented only for the exemplary purpose so asto implement the present invention, and are to be appropriately modifiedor changed depending on the arrangement of an apparatus and variousconditions to which the present invention is applied. Hence, the presentinvention is not limited to the following embodiments. The followingembodiments will explain an example in which the present invention isapplied to an image processing apparatus which includes an imagecapturing unit, and is represented by a digital still camera, digitalvideo camera, and the like.

First Embodiment

FIG. 1 is a block diagram showing the arrangement of an image processingapparatus common to respective embodiments of the present invention.Since this embodiment assumes correction of a backlight scene, FIG. 2shows an example of the backlight scene assumed in this embodiment. Asshown in FIG. 2, in the backlight scene, a background is especiallybrighter than a main object. Respective blocks will be described indetail below with reference to FIGS. 1 and 2.

An exposure determination unit 101 determines an exposure amount uponshooting an input image. This embodiment does not particularly designatea determination method, and uses, for example, an evaluation photometrymethod. In the evaluation photometry method, an image is divided intopredetermined regions, luminance values (luminance signals) calculatedfor respective regions are weighted and averaged using predeterminedweights so as to calculate a representative luminance value of thescene, and an exposure amount is determined based on that luminancevalue.

A reference image capturing unit 102 captures an image using an exposureamount set by the exposure determination unit 101. The image captured bythis unit will be referred to as a reference image hereinafter. When theexposure amount is determined by the aforementioned method, a mainobject is shot to be darker than its actual appearance in the generalbacklight scene shown in FIG. 2.

A luminance-dependent gain generation unit 103 determinescharacteristics of a gain by which the reference image is multiplied.The characteristics of the gain correspond to a gain table with respectto luminance values. In this embodiment, in order to correct a targetbacklight scene, the characteristics of the gain need only be controlledto obtain a bright main object region shown in FIG. 2. Hence, as shownin FIG. 3, luminance-dependent gains are generated to have a pattern toapply a larger gain as a luminance level is lower.

A signal processing unit 104 applies predetermined signal processes tothe reference image. In this embodiment, for example, an opticalcorrection process and noise reduction process are executed as thesignal processes.

<Gain Image Generation Unit 105>

A gain image generation unit 105 generates a gain by which an image isto be multiplied from the luminance-dependent gain table calculated bythe luminance-dependent gain generation unit 103. Gain data, which isgenerated by the gain image generation unit 105 using theluminance-dependent gain table, and is mapped by associating gains withluminance values of respective pixels in an image, will be referred toas a gain image hereinafter. In this embodiment, an input image to thegain image generation unit 105 uses a luminance image generated from animage output from the signal processing unit 104, and the input imageand an image, which is obtained by reducing the input image to leaveonly low-frequency components, are used in gain generation. The reasonwhy a low-frequency image is used is to lower the gain sensitivity tofine details of an image, as described in the paragraphs of the relatedart. In the example shown in FIG. 2, a dark main object portion iscontrolled to be brighter, but when a large gain is applied to a locallydark portion of a building in a background, a luminance difference fromthe remaining portion of the building is reduced, thus impairing acontrast of the image (FIG. 4). Therefore, it is preferable to multiplyfine textures such as windows of the building shown in FIG. 2 by auniform gain according to a representative luminance value of the entirebuilding. For this purpose, by generating the low-frequency image of theinput image, since fine textures disappear to set nearly the sameluminance value as a surrounding luminance value, nearly the same gainvalue need only be multiplied, thus maintaining the contrast. For theaforementioned reasons, this embodiment uses the low-frequency image ofthe input image.

FIG. 5 is a block diagram showing processes of the gain image generationunit 105. The processes of the gain image generation unit 105 will bedescribed below with reference to FIG. 5.

An input image 501 is a luminance image generated from an image outputfrom the signal processing unit 104. A reduction processing unit 502(reduced image generation unit) reduces the input image 501. In thisembodiment, a reduction ratio is not particularly limited, but it isdesirably estimated from frequencies of fine texture portions to which again is to be uniformly applied. Also, a reduction method is notparticularly limited.

<Blur Processing Unit 503>

Blur processing units 503 and 504 execute a predetermined blur processfor the input image 501 and an output image of the reduction processingunit 502. The object of the blur process is to further lower the gainsensitivity to fine textures and to maintain the contrast moresatisfactorily. On the other hand, for example, the blur process ispreferably skipped for a boundary between the main object region andbackground region in FIG. 2. As one of reasons, a dodging process isdesired to respectively multiply the main object region and backgroundregion by different gains. As another reason, since a luminancedifference between the two regions is very large in the assumedbacklight scene, if the blur process is applied to the boundary portion,a pseudo edge as the problem in FIG. 25 is generated conspicuously.Therefore, it is required to apply the blur process while preserving anedge with a large luminance difference.

FIG. 6 is a flowchart showing the processes in the blur processing units503 and 504, and FIG. 7 is a view showing the processing contents of theblur processing units 503 and 504.

As shown in FIG. 6, a reference region is set in step S601. In thisembodiment, a set of pixels within a predetermined range from a positionof a pixel to be processed (to be referred to as a pixel of interesthereinafter) in horizontal and vertical directions is defined as areference region.

Next, in step S602, a predetermined pixel in the reference region isselected as a reference pixel, and a difference absolute value betweenthe pixel of interest and reference pixel is calculated. Furthermore, itis determined in step S603 whether or not the difference absolute valeis not more than a predetermined threshold. In this embodiment, assumethat the threshold is set according to the luminance value of the pixelof interest. This is because level differences of pixels to be selectedare different in a low-luminance region and high-luminance region.

In step S604, the reference pixel, which is determined to have thedifference absolute value not more than the threshold, is selected(extracted). In the example shown in FIG. 7, a reference pixel A locatedin a cloud portion in a background is selected, but a reference pixel Blocated in a person portion is not selected with respect to the pixel ofinterest. In this manner, only reference pixels having closer luminancelevels are selected.

It is confirmed in steps S605 and S606 whether or not all pixels in thereference region have been evaluated. If pixels to be evaluated stillremain, the processes of steps S602 to S604 are repeated while updatinga reference pixel until all the pixels are evaluated. Finally, in stepS607, an average value of all the selected pixels is calculated as anoutput pixel value.

The processes of the blur processing unit 503 have been described. Theblur processing unit 503 can obtain an image which has undergone theblur process while preserving edges having large luminance differences.

Luminance→gain conversion processing units 506 and 507 respectivelyconvert luminance images output from the blur processing units 503 and504 into gain images using a luminance-dependent gain table 505calculated by the luminance-dependent gain generation unit 103. That is,for a plurality of images which include a reduced image and havedifferent reduction ratios, gain data are generated.

An enlargement processing unit 508 applies an enlargement process to again image which is converted from the reduced image by theluminance→gain conversion processing unit 507. In this case, theenlargement process enlarges an image to the same size as that of theinput image. That is, when the reduction processing unit 502 executesreduction of 1/N times, the enlargement processing unit 508 executes anenlargement process of N times. Note that the method of the enlargementprocess is not particularly limited.

<Gain Addition Processing Unit 509>

A gain addition processing unit 509 adds a gain image (first gain data)obtained by converting input luminance values into gains and a gainimage (second gain data) obtained by reducing input luminance values torelatively leave only low-frequency components. The respective imageswill be referred to as images of upper and lower layers.

When gain characteristics of the upper and lower layers are simply addedand averaged, as shown in FIG. 25, a pseudo edge is unwantedly generatedin the vicinity of an edge portion having a large luminance differencedepending on the pattern of the gain table. Therefore, a method of theaddition process has to be devised.

FIG. 8 is a block diagram showing processes of the gain additionprocessing unit 509. The operation of the gain addition processing unit509 will be described below with reference to FIG. 8.

Two types of addition processes are applied to an upper layer image 801and lower layer image 802. FIG. 9 shows the addition process in the gainaddition processing unit 509. As shown in FIG. 9, an addition processingunit 1 adds and averages gains at positions of interest (to be referredto as gains of interest hereinafter) of the upper and lower layers(first addition result).

The processing contents of an addition processing unit 2 will bedescribed below. As shown in FIG. 9, addition processing unit 2 uses thegain of interest of the upper layer and that in a surrounding region (tobe referred to as surrounding gains hereinafter) of the lower layer.Note that the surrounding region corresponds to a set of pixels within apredetermined range from the position of interest in upper, lower, andright, and left directions.

An object of the process to be executed by the addition processing unit2 is to reduce a pseudo edge as the problem to be solved by the presentinvention. FIG. 10 shows, based on the same idea as in FIG. 25, patternsof upper and lower layer gains when a pseudo edge is generated near anedge portion, and a processing image of the addition processing unit 2.In the example shown in FIG. 10, a lower layer gain becomes extremelysmall near an edge portion having a large luminance difference.Therefore, when gains are added at a position X, if gains of interestexpressed by squares are added to each other, the adverse effect of thepseudo edge is worsened. Therefore, as a lower layer gain, a rangeseparated by L from the position X in the right and left directions issearched to select a gain having a value closest to the gain of interestof the upper layer in place of a gain at the position of interest. Inthe example shown in FIG. 10, using a gain at a position X-L expressedby a circle, the influence received from an extremely small gain of thelower layer can be reduced (second addition result). With theaforementioned addition process, generation of a pseudo edge caused bymultiplying an extremely different gain near an edge portion can besuppressed.

FIG. 10 describes the one-dimensional example for the sake ofsimplicity. However, in the actual process of the addition processingunit 2, the same process is executed two-dimensionally. FIG. 11 is aflowchart showing the processing contents of the addition processingunit 2.

In step S1101, difference absolute values between the gain of interestof the upper layer and respective surrounding gains (gains ofsurrounding pixels) of the lower layer are calculated. Then, in stepS1102, M surrounding gains are selected in ascending order of absolutedifference value. Note that M is a predetermined value. With thisprocess, as shown in FIG. 10, surrounding gains of the lower layer,which have gain values closer to the gain of interest of the upperlayer, can be selected. Note that this embodiment adopts the method ofselecting the predetermined number of gains in ascending order ofdifference absolute value. Especially, the process for sorting values inascending order requires a large arithmetic scale and large circuitscale at the time of circuit implementation. Therefore, other methods(for example, a method of selecting all gains corresponding todifference absolute values not more than a predetermined threshold) maybe adopted.

In step S1103, the selected M surrounding gains are weighted and added.A weight (coefficient) used in a weighted addition is determined by:

$\begin{matrix}{{W(k)} = \frac{A}{{{{GL}(k)} - {GH}}}} & (1)\end{matrix}$where GL(k) indicates a k-th gain of the selected surrounding gains ofthe lower layer, and W(k) indicates a corresponding weight of weightedaddition. Also, GH indicates the gain of interest of the upper layer,and A is a predetermined constant. As can be seen from equation (1), aweight becomes larger as a difference from the gain of interest of theupper layer is smaller.

Furthermore, after the weights W(k) for the respective surrounding gainsare calculated using equation (1), the weighted addition is executed by:

$\begin{matrix}{{GL}^{\prime} = \frac{\sum\limits_{k = 1}^{M}\left\{ {{W(k)} \times {{GL}(k)}} \right\}}{\sum\limits_{k = 1}^{M}{W(k)}}} & (2)\end{matrix}$where GL′ is a lower layer gain after the weighted addition.

Finally, in step S1104, the weighted and added surrounding gain GL′ ofthe lower layer and the gain GH of interest of the upper layer are addedand averaged. The added and averaged gain value is output from theaddition processing unit 2.

The operation of the addition processing unit 2 required to reduce apseudo edge has been described. However, when this addition method isapplied to the entire image, a gain pattern unwantedly approaches thatof the upper layer, and is consequently sensitive to textures, thusdeteriorating the contrast of the output image. For example, it isdesirable to suppress a pseudo edge using the result of the additionprocessing unit 2 in the vicinity of an edge portion having a largeluminance difference and large gain difference like a location A, asshown in FIG. 12. However, when the result of the addition processingunit 2 is used for a texture portion having a small gain difference likea location B, the contrast of an image unwantedly decreases. When a gaindifference is smaller, a pseudo edge is not visualized. Hence, in thelocation B, a decrease in contrast is desirably suppressed using theresult of the addition processing unit 1.

Based on the above examination, a final addition result is obtained byweighting and adding the result of the addition processing unit 1 andthat of the addition processing unit 2, and a weight used at that timeis preferably determined by a gain difference of the upper and lowerlayers at the position of interest. Therefore, the results of theaddition processing units 1 and 2 are weighed and added by a weightedaddition unit 806, and a weight used at that time is calculated by aweight calculation unit 805. As described above, the weight of theweighted addition is determined by a difference value of the gains ofinterest of the upper and lower layers, and is set to attach moreimportance to the result of the addition processing unit 2 as thedifference value is larger.

The processing contents of the gain addition processing unit 509 havebeen described. An output image 807 is a gain image 510 as a finaloutput of the gain addition processing unit 509, and is similarly afinal output of the gain image generation unit 105.

A gain multiplying unit 106 multiplies an original image, which hasundergone the predetermined signal processes in the signal processingunit 104, by the final gain image (control gain) generated by the gainimage generation unit 105. The output from the gain multiplying unit 106undergoes tone compression to a predetermined output range by a tonecompression unit 107. Note that the compression method includes a gammaconversion process and the like.

Finally, the image which has undergone tone compression by the tonecompression unit 107 is output to an image display unit 108 and imagerecording unit 109.

As described above, according to the processes of this embodiment, whenupper and lower layer images are added, gain values, which are closer tothe gain of interest of the upper layer, are selected from surroundinggains of the lower layer, and are added, thus suppressing an adverseeffect of a pseudo edge caused by use of a low-frequency image.

Also, based on a gain difference at the position of interest, (1) theaddition result 1 which prioritizes contrast maintenance and (2) theaddition result 2 which prioritizes pseudo tone suppression are weightedand added, thus suppressing a decrease in contrast and pseudo tones.

In this embodiment, images having two types of frequency bands aregenerated from the input image in the gain addition process. However,the present invention is not limited to this. For example, images havingthree or more (two types or more of) different frequency bands may begenerated, and may sequentially undergo the addition process.

Second Embodiment

The second embodiment has different processing contents of the gainimage generation unit 105 shown in FIG. 1 compared to the firstembodiment. FIG. 13 is a block diagram showing processes of the gainimage generation unit 105 according to the second embodiment. As shownin FIG. 13, in this embodiment, luminance images undergo an additionprocess (image addition), and a luminance-dependent gain table isapplied to the added luminance image, thus generating a gain image as anoutput.

Advantages of this arrangement include a reduction of an arithmeticscale since only one conversion processing unit which converts luminancevalues into gains is required, and a reduction of a circuit scale uponcircuit implementation of the conversion process. In this embodiment,two types of images having different frequency bands are generated froman input image, and are added. Alternatively, three or more imageshaving different frequency bands may be generated. In this case, inconsideration of the processing of the first embodiment, luminance→gainconversion processing units as many as the number of images to begenerated are required. Therefore, the effects of this embodiment becomegreater as the number of images to be generated increases.

In FIG. 13, a unit having different processing contents from FIG. 5 ofthe first embodiment is a luminance addition processing unit 1306. Theprocessing contents of the luminance addition processing unit 1306 and,especially, a luminance→weight calculation unit 1405 shown in FIG. 14,will be described below.

In the weight setting method in the weight calculation unit 805 of thefirst embodiment, a weight for the output result of the additionprocessing unit 2 is increased as a gain difference value from a gain ofinterest of an upper layer becomes larger. On the other hand, theluminance→weight calculation unit 1405 of this embodiment calculates aweight based on a luminance difference between an upper layer (firstimage) and lower layer (second image) in place of a gain difference.Therefore, as shown in FIG. 15, even when nearly equal luminancedifferences are generated, gain differences are different depending onluminance values, as shown in FIG. 15. In the example of FIG. 15,although luminance differences are nearly equal to each other duringzones A to C, but gain differences are different. Therefore, whether toprioritize an addition process 1 (addition of pixel values of pixels ofinterest) or an addition process 2 is different even for an identicalluminance difference. Hence, upon calculation of a weight, a processwhich takes a gain difference value into consideration based on aluminance difference value has to be executed. In order to enhance theaforementioned effect (reduction of the arithmetic scale), this processis required to be executed by an arrangement as simple as possible. Notethat the addition process 2 of this embodiment is executed by the samemethod as in the addition process 2 of the first embodiment, and thisprocess is applied to a luminance image in this embodiment in place of again image in the first embodiment.

FIG. 16 is a block diagram showing the operation of the luminance→weightcalculation unit 1405. Inputs are luminance values at positions ofinterest (to be referred to as luminance values of interest hereinafter)of upper and lower layers. Initially, a luminance difference valuecalculation unit 1603 calculates a difference absolute value of theluminance values.

<Gain Amplification Ratio Calculation Unit 1604>

As described above, even when luminance difference values are the same,gain difference value may be largely changed depending on a pattern of again table. A gain amplification ratio calculation unit 1604 calculatesa gain amplification ratio as a value to be multiplied by a differenceabsolute value of luminance values so as to absorb this influence.

FIG. 17 shows calculations of gain amplification ratios. As shown inFIG. 17, the luminance-dependent gain table is divided intopredetermined luminance zones, and a gradient of gain values at startand end points of each zone is calculated. In the example shown in FIG.17, a luminance range is divided into three zones, and gainamplification ratios G1, G2, and G3 are calculated. Furthermore, a gainamplification ratio of a zone, to which a smaller luminance value ofinput luminance values of interest of the upper and lower layersbelongs, is adopted as an output of the gain amplification ratiocalculation unit 1604.

FIG. 18 shows the operation of a luminance→weight conversion unit 1605.As shown in FIG. 18, an input of a weight table is a product of aluminance difference value and gain amplification ratio. A weight valuewith respect to an input value is desirably set to use the result of theaddition processing unit 2 as the input value becomes larger.

The weight calculated by the aforementioned process is an output of theluminance→weight calculation unit 1405, and the addition results of theaddition processing units 1 and 2 are weighted and added by an weightedaddition unit 1406 using this weight, thus obtaining an output image.

The processes of the second embodiment, which are different from thoseof the first embodiment, have been described, and other processes arethe same as those of the first embodiment.

Third Embodiment

The third embodiment has different processing contents of the gain imagegeneration unit 105 shown in FIG. 1 compared to the first embodiment.FIG. 19 is a block diagram showing processes of the gain imagegeneration unit 105 according to the third embodiment. As shown in FIG.19, in this embodiment, gain conversion of an input luminance image isexecuted first, and subsequent processes are then executed. That is, areduction processing unit 1904 generates reduced data from gain datagenerated by a luminance→gain conversion processing unit 1903, thusobtaining reduced gain data.

Advantages of this arrangement include a reduction of an arithmeticscale since only one conversion processing unit which converts luminancevalues into gains is required, and a reduction of a circuit scale uponcircuit implementation of the conversion process as in the secondembodiment.

In FIG. 19, units having different processing contents from FIG. 5 ofthe first embodiment are blur processing units 1905 and 1906. An objectof a blur process is to lower gain sensitivity caused by fine textureswhile holding an edge of, for example, a boundary portion between a mainobject and background, as described in the first embodiment. In order toachieve this object by a gain image input in place of a luminance imageinput, a process which takes a luminance difference value intoconsideration based on a gain difference value, and compensates for thisinfluence is required based on the same idea as the gain amplificationratio of the second embodiment.

FIG. 20 is a flowchart showing the sequence of the blur process shown inFIG. 19. Processes different from the blur process of the firstembodiment are steps S2003 and S2004. In step S2003, a luminanceamplification ratio is calculated based on values of a pixel of interestand reference pixel. Note that “pixel value” described in FIG. 20indicates “gain value”.

FIG. 21 shows calculations of luminance amplification ratios. As shownin FIG. 21, a luminance-dependent gain table is divided intopredetermined zones to have a luminance value as an input, and agradient of luminance values at start and end points of each zone iscalculated. Furthermore, a luminance amplification ratio of a zone, towhich a smaller gain value of those of the input pixel of interest andreference pixel belongs, is adopted as an output.

Next, it is confirmed in step S2004 whether or not a product of theluminance amplification ratio and gain difference absolute value is notmore than a predetermined threshold. If the product is not more than thethreshold, the current reference pixel value is selected. The subsequentprocesses are the same as those of the blur process of the firstembodiment.

After that, images which have undergone the aforementioned blur processare added by a gain addition processing unit 1908, thus obtaining a gainimage as a final output.

The processes of the third embodiment, which are different from those ofthe first embodiment, have been described, and other processes are thesame as those of the first embodiment.

Fourth Embodiment

The fourth embodiment has different processing contents of the gainimage generation unit 105 shown in FIG. 1 compared to the firstembodiment. FIG. 22 is a block diagram showing processes of the gainimage generation unit 105 according to the fourth embodiment.

As shown in FIG. 22, compared to the block diagram shown in FIG. 5, aunit corresponding to the reduction processing unit 502 is replaced by aband limiting unit 2202 (band-limited image generation unit), and theenlargement processing unit is replaced by an image size adjustment unit2208. Originally, the process to be executed by the gain imagegeneration unit 105 is to generate a plurality of images having desiredfrequency bands. The reason why the first embodiment uses the method ofreduction and enlargement of images is to mainly attain a reduction ofan arithmetic scale. In this embodiment, a band limitation method in theband limiting unit 2202 is not particularly limited. For example, amethod of using a low-pass filter having desired frequencycharacteristics may be used.

Also, the image size adjustment unit 2208 executes a process foradjusting a size of an image to be input to a gain addition processingunit of the subsequent stage to that of an input image. Therefore, whenthe band limiting unit 2202 has executed a process including image sizeconversion, the image size adjustment unit 2208 executes anenlargement/reduction process for adjusting the size of an image to aninput image size. Note that when the band limiting unit 2202 does notchange an image size, the image size adjustment unit 2208 skips theprocess.

The processes of the fourth embodiment, which are different from thoseof the first embodiment, have been described, and other processes arethe same as those of the first embodiment.

Fifth Embodiment

The fifth embodiment has different processing contents of the gain imagegeneration unit 105 shown in FIG. 1 compared to the second embodiment.FIG. 23 is a block diagram showing processes of the gain imagegeneration unit 105 according to the fifth embodiment.

As shown in FIG. 23, compared to the block diagram shown in FIG. 13, aunit corresponding to the reduction processing unit 1302 is replaced bya band limiting unit 2302, and the enlargement processing unit 1305 isreplaced by an image size adjustment unit 2305. Note that since theseunits are the same as the contents of the fourth embodiment, adescription thereof will not be repeated.

The processes of the fifth embodiment, which are different from those ofthe second embodiment, have been described, and other processes are thesame as those of the second embodiment.

Sixth Embodiment

The sixth embodiment has different processing contents of the gain imagegeneration unit 105 shown in FIG. 1 compared to the third embodiment.FIG. 24 is a block diagram showing processes of the gain imagegeneration unit 105 according to the sixth embodiment.

As shown in FIG. 24, compared to the block diagram shown in FIG. 19, aunit corresponding to the reduction processing unit 1904 is replaced bya band limiting unit 2404, and the enlargement processing unit 1907 isreplaced by an image size adjustment unit 2407. That is, the bandlimiting unit 2404 generates and obtains band limited-data. Note thatsince these units are the same as the contents of the fourth embodiment,a description thereof will not be repeated.

The processes of the sixth embodiment, which are different from those ofthe third embodiment, have been described, and other processes are thesame as those of the third embodiment.

As described above, even in any of the aforementioned embodiments, whentone characteristics are controlled by applying different gains torespective portions of an input image, generation of a pseudo edge canbe suppressed while satisfactory maintaining the contrast of an outputimage.

In any of the aforementioned embodiments, the image processing apparatusincluding the image capturing unit has been exemplified. However, thepresent invention is not limited to this, and is applicable to otherimage processing apparatuses as long as they execute a tone processusing information of object regions.

Other Embodiments

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-025897, filed Feb. 13, 2013, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: one ormore computer processors configured to implement a reduced imagegeneration unit, a gain data generation unit and a determination unit,wherein: the reduced image generation unit is configured to obtain areduced image by reducing an input image; the gain data generation unitis configured to obtain gain data by associating luminance values forrespective pixels with gains in association with a plurality of images,which include the reduced image and have different reduction ratios; andthe determination unit is configured to determine a control gainrequired to amplify the input image based on the generated gain data,and wherein the determination unit determines the control gain based ona first addition result as an addition result of gains corresponding topixels of interest of first gain data and second gain data, and a secondaddition result as an addition result of a gain corresponding to thepixel of interest of the first gain data, and a gain based on gainscorresponding to pixels of a surrounding region of the pixel of interestin the second gain data, wherein the first gain data for a first imageof the plurality of images and the second gain data is data for a secondimage having a lower frequency than the first image.
 2. The apparatusaccording to claim 1, further comprising an image blurring unitconfigured to execute a blur process while preserving an edge for eachof the plurality of images, wherein said image blurring unit extracts,from pixels in a predetermined range around a pixel of interest, onlypixels having absolute values of differences from the pixel of interest,which are not more than a predetermined threshold, and outputs anaverage value of the extracted pixels, wherein the image blurring unitis implemented by the one or more computer processors.
 3. An imageprocessing apparatus comprising: one or more computer processorsconfigured to implement a gain data generation unit, a reduced datageneration unit and a determination unit, wherein; the gain datageneration unit configured to obtain gain data based on an input image;the reduced data generation unit configured to obtain reduced gain databy reducing the generated gain data; and the determination unitconfigured to determine a control gain required to amplify the inputimage based on a plurality of gain data which include the reduced gain-and have different reduction ratios, wherein the determination unitdetermines the control gain based on a first addition result as anaddition result of gains corresponding to pixels of interest of firstgain data and second gain data, and a second addition result as anaddition result of a gain corresponding to the pixel of interest of thefirst gain data, and a second addition result as an addition result of again corresponding to the pixel of interest of the first gain data and again based on gains corresponding to pixels in a surrounding region ofthe pixel of interest in the second gain data, wherein the first gaindata is a first data of the plurality of gain data and the second gaindata is a second data having a lower frequency than the first data. 4.The apparatus according to claim 3, wherein said determination unitdetermines the control gain by performing weighted addition result andthe second addition result, and when said determination unit performsthe weighted addition of the first addition result and the secondaddition result, said determination unit increases a weight for thefirst addition result as a difference value of gains corresponding tothe pixels of interest of the first gain data and the second gain datais smaller, and increases a weight for the second addition result as thedifference value is larger.
 5. An image processing apparatus comprising:one or more computer processors configured to implement a gain datageneration unit, a band-limited data generation unit and a determinationunit, wherein; the gain data generation unit configured to obtain gaindata based on an input image; the band-limited data generation unitconfigured to generate band-limited data having a frequency banddifferent from the generated gain data; and the determination unitconfigured to determine a control gain required to amplify the inputimage based on a plurality of gain data including the band-limited data,wherein the determination unit determines the control gain based on afirst addition result as an addition result of gains corresponding topixels of interest of first gain data and second gain data, and a secondaddition result as an addition result of a gain corresponding to thepixel of interest of the first gain data, and a gain based on gainscorresponding to pixels in a surrounding region of the pixel of interestin the second gain data, wherein the first gain data is a first data ofthe plurality of gain data and the second gain data is a second datahaving a lower frequency than the first data.
 6. The apparatus accordingto claim 1, wherein said determination unit determines the control gainby performing weighted addition of the first addition result and thesecond addition result, and when said determination unit performs theweighted addition of the first addition result and the second additionresult, said determination unit increases a weight for the firstaddition result as a difference value of gains corresponding to thepixels of interest of the first gain data and the second gain data issmaller, and increases a weight for the second addition result as thedifference value is larger.
 7. An image processing method comprising: areduced image generation step of obtaining a reduced image by reducingan input image; a gain data generation step of obtaining gain data byassociating luminance values for respective pixels with gains inassociation with a plurality of images, which include the reduced imageand have different reduction ratios; and a determination step ofdetermining a control gain required to amplify the input image based onthe generated gain data, wherein in the determination step, the controlgain is determined based on a first addition result as an additionresult of gains corresponding to pixels of interest of first gain dataand second gain data, and a second addition result as an addition resultof a gain corresponding to the pixel of interest of the first gain data,and a gain based on gains corresponding to pixels of a surroundingregion of the pixel of interest in the second gain data, wherein thefirst gain data for a first image of the plurality of images, and thesecond gain data is data for second image having a lower frequency thanthe first image.